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(57) Abstract 

As information to be processed at an object-based video or audio-visual (AV) terminal, an object-oriented bitstream includes 
obiects composition information, and scene demarcation information. Such bitstream stnicture allows on-line editmg. e.g. cut and paste, 
insertion/deletion, grouping, and special effects. In the interest of ease of editing, AV objects and their composition mfonmati on are 
^^sr^i^^^^^^ channels (LCs). Objects which have a lifetime in the decoder beyond their .mtial P^^enmt,^^^ 

time are cached for n^use until a selected expiration time. Hie system includes a de-multiplexer (1), a controller (2) which contro s the 
^JlL of the AV terminal, input buffers (3). AV objects decoders (4), buffers (4') for decoded data, a composer (5). a display (6), and 
an object cache (7). 
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OBJECT-BASED AUDIO-VISUAL TERMINAL 
AND BITSTREAM STRUCTURE 

Tfirhnical Field 

This invention relates to the representation, 
transmission, processing and display of video and audio- 
visual information, more particularly of object-based 
5 information. 

R;=3rkaround of the Invention 

Image and video compression techniques have 
been developed which, unlike traditional waveform coding, 
attempt to capture high-level structure of visual 

10 content. Such structure is described in terms of 
constituent "objects" which have immediate visual 
relevancy, representing familiar physical objects, e.g. 
a ball, a table, a person, a tune or a spoken phrase. 
Objects are independently encoded using a compression 

15 technique that gives best quality for each object. The 
compressed objects are sent to a terminal along with 
composition information which tells the terminal where to 
position the objects in a scene. The terminal decodes 
the objects and positions them in the scene as specified 

20 by the composition information. In addition to yielding 
coding gains, object -based representations are beneficial 
with respect to modularity, reuse of content, ease of 
manipulation, ease of interaction with individual image 
components, and integration of natural, camera -captured 

25 content with synthetic, computer -generated content. 

f^nmmarv of the Invention 

In a preferred architecture, structure or 
format for information to be processed at an object -based 
video- or audio-visual (AV) terminal, an object-oriented 
30 bitstream includes objects, composition information, and 
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scene demarcation information. The bitstream structure 
allows on-line editing, e.g. cut and paste, 
insertion/deletion, grouping, and special effects. 

In the preferred architecture, in the interest 
5 of ease of editing, AV objects and their composition 
information are transmitted or accessed on separate 
logical channels (LCs) . The architecture also makes use 
of "object persistence", taking advantage of some objects 
having a lifetime in the decoder beyond their initial 
10 presentation time, until a selected expiration time. 

Brief Description of the Drawing 

Fig. 1 is a functional schematic of an 
exemplary object-based audio-visual terminal. 

Fig. 2a is a schematic of an exemplary object- 
15 based audio-visual composition packet. 

Fig. 2b is a schematic of an exemplary object - 
based audio-visual data packet . 

Fig. 2c is a schematic of an exemplary compound 
composition packet. 
20 Fig. 3 is a schematic of exemplary node and 

scene description information using composition. 

Fig. 4 is a schematic of exemplary stream-node 
association information . 

Fig. 5 is a schematic of exemplary node/graph 
25 update information using a scene. 

Fig. 6 is a schematic of an exemplary audio- 
visual terminal design. 

Fig. 7 is a schematic of an exemplary audio- 
visual system controller in the terminal according to 
30 Fig. 6. 

Fig. 8 is a schematic of exemplary information 
flow- .in the controller according to Fig. 7. 



2 - 
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Dptailed Description 

An audio-visual (AV) terminal is a systems 
component which is instrumental in forming, presenting or 
displaying audio-visual content. This includes (but is 
5 not limited to) end-user terminals with a monitor screen 
and loudspeakers, as well server and mainframe computer 
facilities in which audio-visual information is 
processed. In an AV terminal, desired functionality can 
be hardware-, firmware- or software -implemented. 

10 Information to be processed may be furnished to the 
terminal from a remote information source via a 
telecommunications channel, or it may be retrieved from a 
local archive, for example. An object-oriented audio- 
visual terminal more specifically receives information in 

15 the form of individual objects, to be combined into 

scenes according to composition information supplied to 
the terminal . 

Fig. 1 illustrates such a terminal, including a 
de-multiplexer (DMUX) 1 connected via a logical channel 

2 0 LCO to a system controller or "executive" 2 and via 

logical channels LGl through LCn to a buffer 3. The 
executive 2 and the buffer 3 are connected to decoders 4 
which in turn are connected to a composer unit 5. Also, 
the executive 2 is connected to the composer unit 5 
25 directly, and has an external input for user interaction, 
for example. 

In the preferred AV architecture, the AV 
objects and their composition information are 
transmitted or accessed on separate logical channels. 

3 0 The DMUX receives the Mux2 layer from the lower layers 

and de-multiplexes it into logical channels. LCO carries 
composition information which is passed on to the 
executive. The AV objects received on other logical 
channels are stored in the buffer to be acted upon by the 
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decoders. The executive receives the composition 
information, which includes the decoding and presentation 
time stamps, and instructs the decoders and composer 
accordingly. 

5 The system handles object composition packets 

(OCP) and object data packets (ODP) . A composition 
packet contains an object's ID, time stamps and the 
"composition parameters" for rendering the object. An 
object data packet contains an object ID, an expiration 
10 time stamp in case of persistent objects, and object 
data. 

Preferably, any external input such as user 
interaction is converted to OCP and/or ODP before it is 
presented to the executive. There is no need for headers 
15 in a bitstream delivered over a network. However, 

headers are required when storing an MPEG4 presentation 
in a file . 

Figs. 2a and 2b illustrate the structure of 
composition and data packets in further detail. Relevant 

20 features are as follows: 

Object ID is composed of object type and object 
number. The default length of the Object ID is 2 bytes, 
including ten bits for the object number and 6 for the 
object type (e.g. text, graphics, MPEG2 VOP, compound 

2 5 object) . An extensible code is used to accommodate more 
than 1023 objects or more than 31 object types. The 
following convention will be adhered to: a value of 
Obllllll in the first six bits of the Object ID 
corresponds to 31 plus the value of the byte immediately 

3Q following the ObjectID; a value of Obll . 1111 . 1111 in the 
least significant 10 bits of the Object ID corresponds to 
1023 plus the value of the two bytes immediately 
following the Object ID (without counting the object type 
extension bytes, if present) . The following object types 
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are defined: 

Composition Objects (16-bit object IDs) 

0X0000 scene configuration object 

0X0001 node hierarchy specification 

5 0X0002 stream-node association 

0X0003 node /scene update 

0X0004 compound object 

Object Data (object type, 6 most significant bits) 
ObOO.OOlO text 
10 ObOO.OOll MPEG2 VOP (rectangular VOP) 

Persistent Objects (PO) are objects that should be 
saved at the decoder for use at a later time. An 
expiration time stamp (ETS) gives the life of a PO in 
milliseconds. A PO is not available to the decoder after 
15 ETS runs out. When a PO is to be used at a later time in 
a scene, only the corresponding composition information 
needs to be sent to the AV terminal. 

Decoding Time Stamp (DTS) indicates the time an 
object (access unit) should be decoded by the decoder, 
20 Presentation Time Stamp (PTS) indicates the time an 

object (access unit) should be presented by the decoder. 

Lifetime Time Stamp (LTS) gives the duration (in 
milliseconds) an object should be displayed in a scene. 
LTS is implicit in some cases, e.g. in a video sequence 
25 where a frame is displayed for 1/frame-rate or until the 
next frame is available, whichever is larger. An 
explicit LTS is used when displaying graphics and text. 
An AV object should be decoded only once for use during 
its life time. 

30 Expiration Time Stamp (ETS) is specified to support 

the notion of object persistence. An object, after it is 
presented, is saved at the decoder (cache) until a time 
given by ETS. Such an object can be used multiple times 
before ETS runs out. A PO with an expired ETS is no 
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longer available to the decoder. 

Object Time Base (0TB) defines the notion of time of 
a given AV object encoder. Different objects may belong 
to different time bases. The AV terminal adapts these 
5 time bases to the local one, as specified in the MSDL VN. 

Object Clock Reference (OCR) can be used if 
necessary to convey the speed of the 0TB to the decoder. 
By this mechanism, OTBs can be recovered/adapted at the 
AV terminal. 

10 Composition Parameters are used to compose a scene 

(place an object in a scene) . These include displacement 
from the upper left comer of the presentation frame, 
rotation angles, zooming factors, etc. 

Priority indicates the priority of an object for 

15 transmission, decoding, and display. MPEG4 supports 32 
levels of priority. Lower numbers indicate higher 
priorities. 

Persistence Indicator (PI) indicates whether an 
object is persistent. 

20 Continuation Indicator (CI) indicates the end of an 

object in the current packet (or continuation) . 

Object Grouping facilitates operations to be applied 
to a set of objects with a single operation. Such a 
feature can be used to minimize the amount of composition 

25 information sent, as well as to support hierarchical 

scene composition based on independent sub-scenes. The 
composer manipulates the component objects as a group. 
The structure of a compound composition packet (CCP) is 
shown in Fig. 2c. 

3 0 Bitstream Structure includes object composition 

packets for describing the composition and controlling 
the presentation of those packets, and object data 
packets that contain the data for the objects, A scene 
is composed by a set of composition packets. The 
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bitstream supports representation of scenes as a 
hierarchy by using compound composition objects (CCP) , 
also known as node hierarchy. A CCP allows combining 
composition objects to create complex audio- visual 
5 scenes . 

Obi ect -Data Packet : 

ObjectID - min (default) 10 bits 
CI and PI could be combined: 

00 - begin non-persistent 
10 01 - begin persistent 

10 - continuation 

11 - end of object 

Priority: 5 bits, present only if CI/PI is ObOO or ObOl 
ETS: 30 bits, present if CI/Pl is ObOl 
15 For prediction-based video coding, VOP_type is indicated 
by two bits (00 (I), 01 (P), 10 (B), 11 (PB)), 
facilitating editing. 
Ob j ect_data_packet { 

ObjectID 16 bits + any extensions; 

20 CIPI 2 bits 

if (CIPI <= 1) { 
Priority 5 bits 

if (object type is MPEG VOP) 

(any prediction based compression) 
25 VOP_type 2 bits 

. } 

if (CIPI == 1) 

ETS 28 bits 

ObjectData 

30 } 

Object Composition Packet 
Ob j ect_composit ion_packet { 

-.ObjectID 16 bits + any extensions 

OCR_Flag 1 bit 
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Display_Timers_Flag 1 bit 

DTS 30 bits 

if {OCR_Flag) 

OCR 30 bits 

if (Display_Timers_Flag) { 

PTS 3 0 bits 

LTS 28 bits 

) 

Compo s i t i on_p a r ame t e r s ; 



10 } 



Composition Parameterc; are defined in section 2 of 
MSDL Verification Model, MPEG N1483, Systems Working 
Draft V2.0, the disclosure of which is incorporated 
herein by reference. 
15 Composition_parameters ( 

visibility 1 bit 

compos it ion_order 5 bits 

number_of_motion_sets 2 bits 
x_delta_0 12 bits 

20 y_delta_0 12 bits 

for (i = 1; i <= number_of_motion_sets ; i++) { 
x_delta_i 12 bits 

y_delta_i 12 bits 

) 

25 } 

Compound Composition Packf^t 
Compound_composition_packet { 

ObjectID 16 bits 

PTS 30 bits 

30 LTS 28 bits 

Compos ition_parameters 
Object Count 8 bits 

-.for (i = 0; i < ObjectCount; i++) ( 
Object_composition_j)acket ; 
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} 

} 

Scene Configuration Packet (SCP) is used to change 
reference scene width, height, to flush the buffer, and 

5 other configuration functions. The object type for SCPs 
is ObOO.OOOO. This allows for 1024 different 
configuration packets. The object number ObOO . 0000 . 0000 
(object ID 0X0000) is defined for use with flushing the 
terminal buffers. 

0 romposition Control for Buffer Management (Object ,ID. 

AV terminal buffers are flushed using 
Flush_Cache and Scene_Update flags. When using 
hierarchical scene structure, the current scene graph is 
.5 flushed and the terminal loads the new scene from the 
bitstream. Use of flags allows for saving the current 
scene structure instead of flushing it. These flags are 
used to update the reference scene width and height 
whenever a new scene begins. If the Flush_Cache_Flag is 
20 set, the cache is flushed, removing the objects (if any) . 
If Scene_Update_Flag is set, there are two possibilities: 
(i) Flush_Cache-Flag is set, implying that the objects in 
the cache will no longer be used; (ii) Flush_Cache_Flag 
is not set, the new scene being introduced (an editing 
25 action on the bitstream) splices the current scene and 
the objects in the scene will be used after the end of 
the new scene. The ETS of the objects, if any, will be 
frozen for the duration of the new scene introduced. The 
beginning of the next scene is indicated by another scene 
30 configuration packet. 

Scene_conf iguration_packet { 

-ObjectlD 16 bits (OXOOOO) 

Flush Cache_Flag 1 bit 
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Scene_Update_Flag 1 bit 

if (Scene_Update_Flag) { 

ref_scene_width 12 bits 

ref_scene_height 12 bits 

5 } 
} 

Composition Control for Scene Description (Object ID 
QxQOOl) 

A hierarchy of nodes is defined, describing a 
10 scene. The scene configuration packets can also be used 
to define a scene hierarchy that allows for a description 
of scenes as a hierarchy of AV objects. Each node in 
such a graph is a grouping of nodes that groups the 
leaves and/or other nodes of the graph into a compound AV 
15 object. Each node (leaf) has a unique ID followed by its 
parameters as shown in Fig. 3. 

Composition Control for Stream-Node Mapping (Object 
ID 0x0002) 

As illustrated by Fig. 4, table entries 
20 associate the elementary object streams in the logical 
channels to the nodes in a hierarchical scene. The 
stream IDs are unique, but not the node IDs. This 
implies that more than one stream can be associated with 
the same node. 

25 Composition Control for Scene Updates (Object ID 

0x0003) 

Fig. 5 illustrates updating of the nodes in the 
scene hierarchy, by modifying the specific parameters of 
the node. The graph itself can be updated by 
30 adding/deleting the nodes in the graph. The update type 
in the packet indicates the type of update to be 
performed on the graph. 

- • Architectural Embodiment 

The embodiment described below includes an 

- 10 - 
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object-based AV bitstream and a terminal architecture. 
The bitstream design specifies, in a binary format, how 
AV objects are represented and how they are to be 
composed. The AV terminal structure specifies how to 
5 decode and display the objects in the binary bitstream. 
AV Terminal Architecture 

Further to Fig. 1 and with specific reference . 
to Fig. 6; the input to the de-mulciplexer 1 is an 
object-based bitstream such as an MPEG-4 bitstream, 

10 consisting of AV objects and their composition 

information multiplexed into logical channels (LC) . The 
composition of objects in -a scene can be specified as a 
collection of objects with independent composition 
specification, or as a hierarchical scene graph. The 

15 composition and control information is included in LCO . 
The control information includes control commands for 
updating scene graphs, reset decoder buffers etc. 
Logical channels 1 and above contain object date. The 
system includes a controller (or "executive") 2 which 

20 controls the operation of the AV terminal. 

The terminal further includes input buffers 3, 
AV object decoders 4, buffers 4' for decoded data, a 
composer 5, a display 6, and an object cache 7. The 
input bitstream may be read from a network connection or 

25 from a local storage device such as a DVD, CD-ROM or 
computer hard disk. LCO containing the composition 
information is fed to the controller. The DMUX stores 
the objects in LCI and above at the location in the 
buffer specified by the controller. In the case of 

30 network delivery, the encoder and the stream server 

cooperate to ensure that the input object buffers neither 
overflow nor underflow. The encoded data objects are 
stored -in the input data buffers until read by the 
decoders at their decoding time, typically given by an 

- n - 
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associated decoding timestamp. Before writing a data 
object to the buffer, the DMUX removes the timestamps and 
other headers from the object data packet and passes them 
to the controller for signaling of the appropriate 
5 decoders and input buffers. The decoders, when signaled 
by the controller, decode the data in the input buffers 
and store them in the decoder output buffers. The AV 
terminal also handles external input such as user 
interaction. 

10 In the object cache 7, objects are stored for 

use beyond their initial presentation time. Such objects 
remain in the cache even if the associated node is 
deleted from the scene graph, but are removed only upon 
the expiration of an associated time interval called the 

15 expiration time stamp. This feature can be used in 

presentations where an object is used repeatedly. over a 
session. The composition associated with such objects 
can be updated with appropriate update messages. For 
example, the logo of the broadcast ing station can be 

20 downloaded at the beginning of the presentation and the 
same copy can be used for repeated display throughout a 
session. Subsequent composition updates can change the 
position of the logo on the display. Objects that are 
reused beyond their first presentation time may be called 

25 persistent objects . 

System Controller (SO 

The system controller controls decoding and 
playback of bitstreams on the AV terminal. At startup, 
from user interaction or by looking for a session at 

30 default network address, the SC first initializes the 
DMUX to read from a local storage device or a network 
port. The control logic is loaded into the program RAM 
at the -time of initialization. The instruction decoder 
reads the instructions from the program and executes 
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them. Execution may involve, reading the data from the 
input buffers (composition or external data) , 
initializing the object timers, loading or updating the 
object tables to the data RAM, loading object timers, or 
5 control signaling. 

Fig. 7 shows the system controller in further 
detail. The DMUX reads the input bitstream and feeds the 
composition data on LCO to the controller. The 
composition data begins with the description of the first 

10 scene in the AV presentation. This scene can be 

described as a hierarchical collection of objects using 
compound composition packets, or as a collection of 
independent object composition packets. A table that 
associates the elementary streams with the nodes in the 

15 scene description immediately follows the scene 

description. The controller loads the object IDs (stream 
IDs) into object list and render list which are 
maintained in the data RAM. The render list contains the 
list of objects that are to be rendered on the display 

20 device. An object that is disenabled by user interaction 
is removed from the render list. A node delete command 
that is sent via a composition control packet causes the 
deletion of the corresponding object IDs from the object 
list. The node hierarchy is also maintained in the data 

25 RAM and updated whenever a composition update is 
received . 

The composition decoder reads data from the 
composition and external data buffer and converts them 
into a format understood by the instruction decoder. The 
30 external input includes user interaction to select 
objects, disenable and enable objects and certain 
predefined operations on the objects. During the 
execution of the program, two lists are formed in the 
data RAM. The object list, containing a list of objects 
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(object IDs) currently handled by the decoders and a 
render list, containing the list of active objects in the 
scene. These lists are updated dynamically as the 
composition information is received. For example, if a 
5 user chooses to hide an object by passing a command via 
the external input, the object is removed from the render 
list until specified by the user. This is also how 
external input is handled by the system. Whenever there 
is some external interaction, the composition decoder 
10 reads the external data buffer and performs the requested 
operation. 

The SC also maintains timing for each AV object 
to signal the decoders and decoder buffers of decoding 
and presentation time. The timing information for the AV 

15 objects is specified in terms of its time-base. The 
terminal uses the system clock to convert an object's 
time base into system time. For objects that do not need 
decoding, only presentation timers are necessary. These 
timers are loaded with the decoding and presentation 

20 timestamps for that AV object. The controller obtains 
the timestamps from the DMUX for each object. When a 
decoding timer for an object runs out, the appropriate 
decoder is signaled to read data from the input buffers 
and to start the decoding process. When a presentation 

25 timer runs out, the decoded data for that object is 
transferred to the frame buffer for display. A dual 
buffer approach could be used to allow writing to a frame 
buffer while the contents of the second buffer are 
displayed on the monitor. The instruction decoder can 

30 also reset the DMUX or input buffers by signaling a 
reset, which initializes them to the default state. 
Information Flow in the Controller 

Fig. 8 shows the flow of information in the 
controller. To keep the figure simple, the operations 

- 14 - 
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performed by the instruction decoder are shown in groups. 
The three groups respectively concern object property 
modifications, object timing, and signaling. 
Object Property Modifications 
5 These operations manipulate the object IDs, 

also called elementary stream IDs. When a scene is 
initially loaded, a scene graph is formed with the object 
IDs of the objects in the scene. The controller also 
forms and maintains a list of objects in the scene 
10 {object list) and active objects in the object from the 
render list. Other operations set and update object 
properties such as composition parameters when the 
terminal receives a composition packet . 
object Timing 

15 This group of operations deals with managing 

object timers for synchronization, presentation, and 
' decoding. An object's timestamp specified in terms of 
its object time base is converted into system time and 
the presentation and decoding time of that object are 
20 set. These operations also set and reset expiration 
timestamps for persistent objects. 
Signaling 

Signaling operations control the over-all 
operation of the terminal. Various components of the 

25 terminal are set, reset and operated by controller 
signaling. The controller checks the decoding and 
presentation times of the objects in the render list and 
signals the decoders and presentation frame buffers 
accordingly. It also initializes the DEMUX for reading 

30 from a network or a local storage device. At the 

instigation of the -controller, decoders read the data 
from the input buffers and pass the decoded data to 
decoder output buffers. The decoded data is moved to 
the presentation device when signaled by the controller. 
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WE CLAIM: 



1 1. A method for displaying object-based audio- 

2 visual/video data, comprising: 

3 (a) receiving, over time, a plurality of audio- 

4 visual/video objects and composition information for the 

5 objects; 

6 (b) storing in a cache memory at least one of 

7 the objects; 

8 (c) composing scenes from said objects 

9 including the one of the objects stored in the cache 

10 memory; and 

11 (d) displaying the composed scenes . 

1 2. The method of claim 1, further comprising, in 

2 addition to storing the one of the objects, storing 

3 expiration time data for the one of the objects. 

1 3. The method of claim 1, with at least one of the 

2 objects being received from a network connection. 

1 4. The method of claim 1, with at least one of the 

2 objects being received from local memory. 

1 5. The method of claim 1, with at least one of the 

2 objects being received from local memory and at least one 

3 other of the objects being received from a network 

4 connection, and with the composed scenes comprising the 

5 one and the other of the objects. 

1 6. The method of claim 1, further comprising 

2 responding to interactive user input. 

1 7. The method of claim 6, wherein responding 



wo 98/36559 



PCT/US98/02668 



2 comprises at least one of selecting, enabling and 

3 disenabling one of the objects. 

1 8. Apparatus for displaying object -based audio- 

2 visual /video data, comprising: 

3 (a) a controller circuit for controlling 

4 /acquisition over time of a plurality of audio- 

5 visual/video objects and composition information for. the 

6 objects; 

7 (b) a cache memory for storing at least one of 

8 the objects; 

9 (c) a composer circuit, coupled to the cache 

10 memory, for composing scenes from said video objects 

11 including the one of the objects stored in the cache 

12 memory; and 

13 (d) a display for the composed scene. 

1 9. Apparatus for displaying object-based audio- 

2 visual /video data, comprising a processor which is 

3 instructed for: 

4 (a) controlling acquisition over time of a 

5 plurality of audio- visual /video objects and composition 

6 information . for the objects; 

7 (b) storing in a cache memory at least one of 

8 the objects; 

9 (c) composing scenes from said video objects 

10 including the one of the objects stored in the cache 

11 memory; and 

12 (d) displaying the composed scenes. 

1 10. Apparatus for displaying object-based audio- 

2 visual /video data, comprising: 

3 - - (a) means for controlling acquisition over 

4 time of a plurality of audio-visual/video objects and 



wo 98/36559 



PCT/US98/02668 



5 composition information for the objects; 

6 (b) means for for storing in a cache memory at 

7 least one of the objects; 

8 (c) means, coupled to the cache memory, for 

9 composing scenes from said video objects including the 

10 one of the objects stored in the cache memory; and 

11 (d) means for displaying the scene. 
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PCT/US98/02668 



V. Reasoned statement under Rule 66 2(a)(ii) with regard to novelty, inventive step or industrial applicability- 
citations and exolanations sunnnrtino aii^i. *^ 



citations and explanation s supporting such statement 

1. STATEMENT 

Novelty (N) Claims NONE 



Claims 1-10 



Inventive Step (IS) 



Industrial Applicability (lA) 



Claims NONE 



Claims 1-10 



Claims 1-10 



Claims NONE 



YES 
NO 

YES 
NO 



YES 
NO 



2. CITATIONS AND EXPLANATIONS 

Claims i-10 lack novelty under POT Article 33(2) as being anticipated by Mimura et al (5,548,346). 
claim 1: 

by nlTvl2piriTOR. ' °' audicvisual/video ejects and composition infoxmation for the objects 



IS met 



the claimed storing in a cache memory at least one of the objects is met by col 13 lines 3-5 
|he claimed composing scenes from said objeas including the one of the objects stored in the'cache memory is met 
oy "ig. ii Items j, oo. 



claim 2: 



claim 3: 



claim 4: 



claim 5: 



the claimed displaying the composed scenes is met by Fig, 11 item 5, coL 4 lines 55-62. 
the Umitation of claim 2 is inherent. 

the limitation of claim 3 is met by Fig. 22 item 2204, col 4 lines 37-38. 
the limitation of claim 4 is met by Fig, 11 item 12. 



the limitation of claim 5 is met by Fig. 1 1 
items AV SEPARATOR, 12, 3,66. 



claim 6: 



claim 7: 



claim 8: 



the limitation of claim 6 is met by Fig. 11 item 51, Fig. 17 item 51. 



the limitation of claim 7 is inherent. 



the claimed a controller circuit for controlling acquisition (Continued on Supplemental Sheet.) 



Form PCT/IPEA/408 (Box V) (January 1994) ★ 



WRITTEN OPINION 



International application No. 
PCT/US98/02668 



Supplemental Box 

(To be used when the space in any of the preceding boxes is not sufficient) 



Continuation of: Boxes I - VIII 



Sheet 10 



TIME LIMIT: 

The time limit set for response to a Written Opinion may not be extended. 37 CFR 1 484(d) Any response 
received after the expiration of the time Hmit set in the Written Opinion will not be considered in preparing the In^mational 
Preliminary Exammation Report. ^ t-^ ximwuudi 



V. 2. REASONED STATEMENTS - CITATIONS AND EXPLANATIONS (Continued): 

over tune of a plurality of audio-visual/video ojecis and composition infonnaiion for the objects is met by Fie 17 item 1 1 1 
col. 4 lines 4-33. j ^ ixi, 

the claimed a cache memory for storing at least one of the objects is met by coL 13 lines 3-5 
the claimed a composer circuit, coupled to the cache memory, for composing scenes from said video objects 
mcluding the one of the objects stored in the cache memory is met by figs. 4, 5 col. 12. 
the claimed a display for the composed scene is met by col. 4 lines 55-62. 

claim 9: 

claim 9 lacks novelty for the reasons as claim 8. 

claim JO: 

claim 10 lacks novelty for the reasons as claim 8. - 



NEW CITATIONS 



NONE 



Form PCT/IPEAy408 (Supplemental Box) (January 1994) ★ 
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r- • ^Aulhorily or. if Iwo or more Aulhorilies are 



IPEA/ 



PCX 

DEMAND 



\ CHAPTER I1_ 



For international Preliminary Exan^ining Authority use only 




[Tntemational application No. 
PCT/US98/02668 

fei?^SD AUDIO-VISUAL TERMINAL AND B.TSTREAM STRUCTURE 



(Earliest) Priority date (dayMonth/year) 
14 February 1997 ( 14-02-97 ) 



rToxNclI APPUCANT(S) Z^JZiiiZfiiir^ 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF 
Broadway and 1 16th Street 
I New York. NY 10027-6699 

lUS 



State (that is. country) of nationality : 
lUS 



Facsimile No.: 
(212)854-8463 



Teleprinter No.: 



State (that is. country) of residence: 
US 



A ,Hrfress- (Family name foUoy>ed by given 
1 Name and address, fr ^ country.) 

ELEFTHERIADIS, ALEXANDROS 
Department of Electrical Engineenng 
Columbia University 
SOOWest 120th Street 
New York. NY 10027-6699 
US 



. for a legal entity, full official designator,. 



ion. The address must include postal code and 



State (that is. country) of nationality: 
GR 

r== 7THHre«- family name followed by given 

I Name and address. ^^^^ ^^^ournry.; 

KALVA. HARI 

Department of Electrical Engineenng 

Columbia University 

500 West 120th Street 

New York, NY 10027-6699 

US 



State (that is. country) of residence: 




name, 



, for a legal entity, full official designation. 



The address must include postal code and 



State (that is. country) of nationality: 

IN 

Further applicants are indicated on a continuation sheet 



State rr/u^n*. country; of residence: 
US 



□ 



Form 



PCT/IPEAy40l (first sheet) ^uiy i^^^j 



TooalSlar 1998. Form PCTDtM 



See Notes to the demand form 



Sheet No. 



International application No. 

PCT/US98/02668 



Box No. VI CHECK LIST 



The demand is accompanied by the following elements, in the language referred to in 
Box No. IV, for the purposes of international preliminary examination: 



I- translation of international application 

2. amendments under Article 34 

3, copy (or where required, translation) 
of amendments under Article 19 



4. copy (or, where required, translation) 
of statement under Article 19 



5. letter 

6. other (specify) 



sheets 
sheets 

sheets 

sheets 
sheets 
sheets 



For International Preliminary 
Examining Authority use only 

received 



□ 
□ 

□ 

□ 
□ 
□ 



not received 

□ 
□ 

□ 

□ 
□ 
□ 



The demand is also accompanied by the item(s) marked below: 

1. fee calculation sheet 

2. I I separate signed power of attorney 

3. I I copy of general power of attorney; 
' — ' reference number, if any: 



4. I I statement explaining lack of signature 

5. I I nucleotide and or amino acid sequence listing in 
' — I computer readable form 

other (^5pec/5'>': Transmittal Letter 



Box No. VII SIGNATURE OF APPLICANT, AGENT OR COMMON REPRESENTATIVE 



Next to each signature, indicate the name of the person signing and the capacity in which the person signs (if such capacity is not 
obvious from reading the demand). 



Peter A. Businger (Agent). 




For International Preliminary Examining Authority use only ' 
1. Date of actual receipt of DEMAND: 



2. Adjusted date of receipt of demand due 
to CORRECTIONS under Rule 60.1(b): 



3. I I The date of receipt of the demand is AFTER the expiration of 1 9 months r~| 
* — ' from the priority date and item 4 or 5. below, does not apply. I 1 



The applicant has been 
informed accordingly. 



4. I I The date of receipt of the demand is WITHIN the period of 19 months from the priority date as extended by virtue of 
* — ' Rule 80.5. 



5. I I Although the date of receipt of the demand is after the expiration of 19 months from the priority date, the delay in arrival is 
' — • EXCUSED pursuant to Rule 82. 



For International Bureau use only 



Demand received from IPEA on: 



Form PCT/IPEA/401 (last sheet) (July 1998) 



LogalStar 1998. Form PCTOEM 



See Notes to the demand form 



PCX 

FEE CALCULATION SHEET 
Annex to the Demand for international preliminary examination 



CHAPTER II 



Intemaiional 
application No. 


PCT/US98/02668 


Applicant's or agent's 
file reference 


30919-PCT 



For International Preliminar\' Examining Authority use only 



Date stamp of the IPEA 



Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



Calculation of prescribed fees 



1. Preliminary examination fee 

2. Handling fee (Applicants from certain States are 
entitled to a reduction of 75% of the handling fee. 
Where the applicant is (or all applicants are) so 
entitled, the amount to be entered at H is 25% of the 
handling fee.) 

3. Total of prescribed fees 

Add the amounts entered at P and H 

and enter total in the TOTAL box 



490.00 



162.00 



H 



652.00 



TOTAL 



Mode of Payment 

□ authorization to ch£u-ge deposit 
account with the IPEA (see below) 

cheque 
I I postal money order 
I I bank draft 



I I cash 

I I revenue stamps 

I I coupons 

I I other (specify): 



Deposit Account Authorization (this mode of payment may not be available at all IPEAs) 

The IPEA/ | | is hereby authorized to charge the total fees indicated above to my deposit account. 



(this check-box may be marked only if the conditions for deposit accounts of the IPEA so permit) is 
hereby authorized to charge any deficiency or credit any overpi 



02-4377 



above to my deposit account. 
9 September 1998 



Deposit Account Number 



Date (day/month/year) 



overpayment in the total fees indicated 



Signature 




Form PCT/IPEA/401 (Annex) (July 1998) 



Logaistar 1998. Form PCTDFEE 



See Notes to the feeValculation sheet 



#4 



ATENT COOPERATION 



From the 

INTERNATIONAL PRELIMINARY EXAMINING AUTHORITY 



M 1^ ' Per 



To: 

HENRY TANG 
BAKER & BOTTS, LLP 
30 ROCKEFELLER PLAZA 
NEW YORK NY 10112-0228 


per 

NOTIFICATION OF RECEIPT 
OF DEMAND 

(PCT Rule 61, 1(b), first sentence 
and Administrative Instructions, Section 601) 


Date of mailing -i n nPT 100Q 


Applicant's or agent's file reference 

30919-PCT 


IMPORTANT NOTIFICATION 


International application No. 

PCT/US98/02668 


International filing date {day/month/year) 

13 FEB 98 


Priority date (day /month/year) 

14 FEB 97 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF 
NEW YORK 



The applicant is hereby notified that this International Preliminary Examining Authority considers the following date as the 
date of receipt of the demand for international preliminary examination of the international application: 



09 SEP 1998 




2. This date of receipt is: RECEIVED 

BAKER & BOTTS, L.L.P. 

ryi the actual date of receipt of the demand. . . 

98 OCT 19 PM 3* 2!5 

I j die date on which the proper corrections to the demand were timely received. / * • 

TO 

3. j I This date is AFTER the expiration of 19 months from the priority date. 

Attention: The election(s) made in the demand does (do) not have the effect of postponing the commencer J>nt of the' 
national phase until 30 months from the priority date (or later in some Offices) (Article 39(1)). Therefore, trie acts for 
entry into the national phase must be performed within 20 months from the priority date (or later in some Offices) 
(Article 22). 

For details, see Annex B to Form PCT/IB/301 sent by the International Bureau and Volume 11 of the PCT Applicant's 
Guide. 

I j This notification confirms die information given in person or by telephone on: 



4. Only where paragraph 3 applies, a copy of diis notification has been sent to die International Bureau. 





Name and mailing address of the IPEA/US 
Assistant Commissioner for Patents 
Box PCT 

Washington. D.C. 20231 Atm: IPEA/US 
Facsimile No. 


Authorized officer '\ 
Telephone No. 




Form PCT/IPEA/402 (July 1992) 




KET FOR 






PATENT COOPERATION TREATY 



PCT/US98/02668 



BAKER & BO ilS.a ir^ 



From the INTERNATIONAL$ftJl^l^l*2PM12 



PCT 



INFORMATION CONCERNING ELECTED 
OFFICES NOTIFIED OF THEIR ELECTION 

(PCT Rule 61.3) 



Date of mailing (day/month/year) 
22 October 1998 <22.10.98) 



To: 



TANG, Henry 
Baker & Botts, LLP 
30 Rockefeller Plaza 
New York, NY 101 12-0228 
ETATS-UNIS D'AMERIQUE 




Applicant's or agent* s file reference 
30919-PCT 


IMPORTANT INFORMATION 


international appUcatiori No. 

PCT/US98/02668 


International filing date (day/month/year) 

13 February 1998 (13.02.98) 


Priority date (day/month/year) 

14 February 1997 (14.02.97) 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK et al 



1. The applicant is hereby informed that the International Bureau has, according to Article 31(7), notified each of the following 
Offices of its election: 

EP :AT,BE,CH,DE,DK,ES,FI,FR,GB,GRJE,IT,LU,MC,NUPT,SE 
Natior^al :CA,JP,US 

2. The following Offices have waived the requirement for the notification of their election; the notification will be sent to them 
by the International Bureau only upon their request: 

None 

3 The applicant is reminded that he must enter the "national phase" before the expiration of 30 months from the pnority date 
before each of the Offices listed above. This must be done by paying the national fee(s) and furnishing , if prescribed, a 
translation of the International application (Article 39(l)(a)), as well as, where applicable, by furnishing a translation of any 
annexes of the international preliminary examination report (Article 36(3)(b) and Rule 74.1). 

Some offices have fixed time limits expiring later than the above-mentioned time limit. For detailed information about the 
applicable time limits and the acts to be performed upon entry Into the national phase before a particular Office, see Volume I 
of the PCT Applicant's Guide. 

The entry into the European regional phase Is postponed until 31 months from the priority date for all States designated for 
the purposes of obtaining a European patent. 



.oi DOC ;t fo^ 



The International Bureau of WlPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 



Facsimile No. (41-22) 740.14.35 



Authorized officer:^^-,^ 

J. Leitao 

Telephone No. (41-22) 338.83.38 



Form PCT/IB/332 (September 1997) 



2299228 



PATENT COOPERATION TREATY 

From the INTERNATIONAL BUREAU 



PCT/US98/02668 



PCT 



NOTIFICATION CONCERNING 
SUBMISSION OF PRIORITY DOCUMENTS 

(PCT Administrative Instructions, Section 41 1) 



Date of mailing (day/month/year) 
31 March 1998 (31.03.98) 



To: 



RECEIVED 
BAKER & BOTTS, L.L.P. 

TANG, Henry 98 APR 1 1| AM IG: 39 

Baker & Botts, LLP 

30 Rockefeller Plaza jq 
New York, NY 1 01 1 2-O228-77 
ETATS-UNIS D'AMERIQ=*& 




Applicant's or agent's file reference 
30919-PCT 


IMPORTANT NOTIFICATION 


International application No. 
PCT/US98/02668 


International filing date (day/month/year) 

13 February 1998 {13.02.98) 


Priority date (day/month/year) 

14 February 1997 (14.02.97) 


Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK et al 



The applicant is hereby notified of the date of receipt by the International Bureau of the priority document(;>) relating to the 
following apptication(s): 



Priority application No: Priority date: Priority country: Date of receipt of priority document: 

60/037,779 14 Feb 1997 (14.02.97) US 19 Mar 1998 (19.03.98) 





Authorized officer m 


The International Bureau of WlPO 




34, chemin des Colombettes 


F. Zotomayor 


1211 Geneva 20, Switzerland 




Facsimile No.: {41-22)740.14.35 


Telephone No.: (41-22)338.83.38 



Form PCT/IB/304 (July 1992) 



001967273 



^3 



wo 98/36559 
PCT/US98/02668 



r-»ATENT COOPERATION TRi?ATY /Cj 

From the INTERNATIONAL BUR|/\Mr|vc>fi^>/- 



PCT 



NOTICE INFORMING THE APPLICANT OF THE 
COMMUNICATION OF THE INTERNATIONAL 
APPLICATION TO THE DESIGNATED OFFICES 

(PCT Rule 47.1(c), first sentence) 



Date of mailing (day/momh/year) 
20 August 1998 (20.08.98) 



To: BAKER&30TTS, L.L.P. 

TANG, Henry 

Baker & Botts, LLP 98 SEP " 1 PM 12= 55 
30 Rockefeller Plaza 
New York, NY 10112-0228 TG 
ETATS-UNIS D'AMERIQ 




Applicant's or agent's file reference 
30919-PCT 



IMPORTANT NOTICE 



International application No. 

PCT/US98/02668 



International filing date (day/moath/year) 
13 February 1998 (13.02.98) 



Priority date (day/month/year) 

14 February 1997 (14.02.97) 



Applicant 



THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK et al 



1. Notice is hereby given that the International Bureau has communicated, as provided in Article 20, the international application 
to the following designated Offices on the date indicated above as the date of mailing of this Notice: 

CA,EP,JP,US 

In accordance with Rule 47.1(c), third sentence, those Offices will accept the present Notice as conclusive evidence that 
the communication of the international application has duly taken place on the date of mailing indicated above and no copy 
of the international application is required to be furnished by the applicant to the designated Office(s). 

2. The following designated Offices have waived the requirement for such a communication at this time: 

None 

The communication will be made to those Offices only upon their request. Furthermore, those Offices do not require the 
applicant to furnish a copy of the international application (Rule 49.1(a-bis)). 




3. Enclosed with this Notice is a copyj>Hhe international application as published by the International Bureau on 
20 August 1 998 (20.08.98) under No. WO 98/36559 

REMINDER REGARDING CHAPTER II (Article 31(2)(a) and Rule 54.2) 

If the applicant wishes to postpone entry into the national phase until 30 months (or later in some Offices) from the priority 
date, a demand for international preliminary examination must be filed with the competent International Preliminary 
Examining Authority before the expiration of 19 months from the priority date. 

It is the applicant's sole responsibility to monitor the 19-month time limit. 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has the 
right to file a demand for international preliminary examination. 

REMINDER REGARDING ENTRY INTO THE NATIONAL PHASE (Article 22 or 39(1)) 

If the applicant wishes to proceed with the international application in the national phase, he must, within 20 months 
or 30 months, or later in some Offices, perform the acts referred to therein before each designated or elected Office. 

For further important information on the time limits and acts to be performed for entering the national phase, see the 
Annex to Form PCT/IB/301 (Notification of Receipt of Record Copy) and Volume II of the PCT Applicant's Guide. 



■ ^ 


r / ^7//j>- ■ 


The Irrternational Bureau of WlPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 

Facsimile No. (41-22) 740.14.35 


Authorizes pfficer 

J. Zahra 

Telephone No. (41-22) 338.83.38 



Form PCT/IB/308 (July 1996) 



2194216 



^^ATENT COOPERATION TREATY 



PCT/US98/02668 
397/^ 



From the INTERNATIONAL BUREAU 



PCT 



NOTIFICATION OF RECEIPT OF 
RECORD COPY 

(PCT Rule 24.2(a)) 



To: 



RECEIVED 
BAKER & BOTTS. L.L.P 



TANG, Henry 
Baker & Botts, LLP 
30 Rockefeller Plaza 
New York, NY 10112-0228 
ETATS-UNIS D'AMERIQUE 



98 APR -6 AM 10: 28 



L I \N, 



Date of mailing (day/month/year) 
23 March 1998 (23.03.98) 


IMPORTANT NOTIFICATION 


Applicant's or agent's file reference 
30919-PCT 


International application No. 
PCT/US98/02668 



The applicant is hereby notified that the International Bureau has received the record copy of the international application as 
detailed below. 

Name(s) of the applicant(s) and State(s) for which they are applicants: 

THE TRUSTEES OF COLUMBI_A UNIVERSITY IN THE CITY OF NEW YORK (for all designated 
States except US) 

ELEFTHERIADIS, Alexandres et al (for US) 

International filing date 1 3 February 1 998 (1 3.02.98) 

Priority date(s) claimed 14 February 1997 (14.02.97) 

Date of receipt of the record copy 
by the International Bureau : 

List of designated Offices 



17 March 1998 (17.03.98) 



EP :AT,BE,CH,DE,DK,ES,FI,FR,GB,GR,IE,IT,LU,MC,NL,PT,SE 
National :CA,JP,US 



ATTENTION 

The applicant should carefully check the data appearing in this Notification. In case of any discrepancy between these data 
and the indications in the international application, the applicant should innmediately inform the International Bureau. 

In addition, the applicant's attention is drawn to the information contained in the Annex, relating to: 



time limits for entry into the national phase; 
I X I confirmation of precautionary designations; 
I I requirements regarding priority documents. 



A copy of this Notification is being sent to the receiving Office and to the International Searching Authority. 






The international Bureau of WlPO 
34, chemin des Colombettes 
1211 Geneva 20, Switzerland 

Facsimile No. (41-22)740.14.35 


Authorized officer: 

J. Leitao 

Telephone No. (41-22)338.83.38 



Form PCT/IB/301 (September 1996) 



001955715 



O FORM PCT/IB/301 



International application No. 

PCT/US98/02668 



INFORMATION ON TIME LIMITS FOR ENTERING THE NATIONAL PHASE 

The applicant is reminded that the "national phase" must be entered before each of the designated Offices indicated in the 
Notification of Receipt of Record Copy (Form PCT/IB/301) by paying national fees and furnishing translations, as prescribed by 
the applicable national laws. 

The time limit for performing these procedural acts is 20 MONTHS from the priority date or, for those designated States 
which the applicant elects in a demand for international preliminary examination or in a later election, 30 MONTHS from the 
priority date, provided that the election is made before the expiration of 1 9 months from the priority date. Some'designated (or 
elected) Offices have fixed time limits which expire even later than 20 or 30 months from the priority date. In other Offices an 
extension of time or grace period, in some cases upon payment of an additional fee, is available. 

In addition to these procedural acts, the applicant may also have to comply with other special requirements applicable in 
certain Offices, it is the applicant's responsibility to ensure that the necessary steps to enter the national phase are taken in a 
timely fashion. Most designated Offices do not issue reminders to applicants in connection with the entry into the national 
phase. 

For detailed information about the procedural acts to be performed to enter the national phase before each designated 
OfTice, the applicable time limits and possible extensions of time or grace periods, and any other requirements, see the relevant 
Chapters of Volume 11 of the POT Applicant's Guide. Information about the requirements for filing a demand for international 
preliminary examination is set out in Chapter IX of Volume I of the POT Applicant's Guide. 



GR and ES became bound by PCT Chapter II on 7 September 1996 and 6 September 1997, respectively, and may, therefore, 
be elected in a demand or a later election filed on or after 7 September 1996 and 6 September 1997, respectively, regardless of 
the filing date of the international application. (See second paragraph above,) 

Note that only an applicant who is a national or resident of a PCT Contracting State which is bound by Chapter II has 
the right to file a demand for international preliminary examination. 



CONFIRMATION OF PRECAUTIONARY DESIGNATIONS 

This notification lists only specific designations made under Rule 4.9(a) in the request. It is important to check that these 
designations are correct. Errcs 'n designations can be corrected where precautionary designations have been made under 
Rule 4.9(b). The applicant is hereby reminded that any precautionary designations may be confirmed according to Rule 4.9(c) 
before the expiration of 15 months from the priority date. If it is not confirmed, it will automatically be regarded as withdrawn 
by the applicant. There will be no reminder and no invitation. Confirmation of a designation consists of the filing of a notice 
specifying the designated State concerned (with an indication of the kind of protection or treatment desired) and the payment 
of the designation and confirmation fees. Confirmation must reach the receiving Office within the 15-month time limit. 

REQUIREMENTS REGARDING PRIORITY DOCUMENTS 

For applicants who have not yet complied with the requirements regarding priority documents the following is recalled. 

Where the priority of an earlier national (i.e., national or regional) application is claimed, the applicant must submit a copy 
of the said national application, certified by the authority with which it was filed ("the priority document") to the receiving Office 
(which will transmit it to the International Bureau) or directly to the International Bureau, before the expiration of 16 months from 
the priority date (Rule 17.1). 

Where the priority document is issued by the receiving Office, the applicant may. Instead of submitting the priority 
document, request the receiving Office to prepare and transmit the priority document to the International Bureau. Such 
request must be made before the expiration of the 1 6-month time limit. 

It is recalled that, where several priorities are claimed, the priority date to be considered for the purposes of computing the 
16-month time limit is the filing date of the earliest application whose priority is claimed. 

If the priority document concerned is not submitted to the International Bureau before the expiration of the 16-month time 
limit, or if the request to the receiving Office to transmit the priority document has not been made (and the corresponding fee, if 
any, paid) before the expiration of this time limit, any designated State may disregard the priority claim. 



Form PCT/IB/301 (Annex) (September 1997) 



001955715 



#c»r 



REQUEST 



The undersigned requests that the present 

international application be processed 
according to the Patent Cooperation Treaty. 



receiving Office use only 



InterDational Application No, 



International Filing Date 



Nanic of receiving OfRcc and "PCT International Application" 



Applicant's or agent's file reference 



Box No. I TITLE OF INVENTION 

OBJECT-BASED AUDIO-VISUAL TERMINAL AND BITSTR 


EAM STRUCTURE 


Box No. n APPLICANT 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (i.e. country) of residence if no State of residence is indicated below.) 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 
Broadway and 116th Street 
New York. NY 10027-6699 
US 


1 1 This person is also inventor. 


Telephone No. 
(212) 854-8444 


Facsimile No. 
(212) 864-8463 


Teleprinter No. 


State (i.e. country) of nationality: 
US 


State f7.e. country) of residence: 
US 


piis person is applicant 1 1 all designated all designated States except 1 1 the United States | 1 the States indicated in 
for the purposes of: 1 1 States ICy the United States of America 1 1 of America only 1 1 the Supplemental Box 


Box No. ni FURTHER APPLICANT(S) AND/OR (FURTHER) INVENTOR(S) 


Name and address: (Family name follon-ed by given name; for a legal entity, full official desigruition. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (i.e. country) of residence if no State of residence is indicated below.) 

ELEFTHERIADIS. ALEXANDROS 

Department of Electrical Engineering 

Columbia University 

500 West 120th Street 

NewYoric. NY 10027-6699 

US 


This person is: 

1 1 applicant only 

applicant and inventor 

1 1 inventor only (If this check-box 
is marked, do not fill in below.) 


State (i.e. country) of nationality: 
GR 


State (Le. country) of residence: 
US 


This person is applicant 1 1 all designated 1 1 all designated States except ^\ the United States 1 1 the States indicated in 
for the purposes of: 1 1 States 1 — 1 the United States of America of America only 1 1 the Supplemental Box 


XI Further applicants and/or (further) inventors are indicated on a continuation sheet. 


Box No. I\ AGENT OR COMMON REPRESENTATIVE; OR ADDRESS FOR CORRESPONDENCE 


The person identified below is hereby/has been appointed to act on behalf |^| i — \ 

of the applicant(s) before the competent International Authorities as: l^vl ag^nx | | common representative 


Name and address: (Family name followed by given name: for a legal entity, full official 
designation. The address must include postal code and name of country.) 

TANG. HENRY and 

BUSINGER. PETER A. 

Baker & Botts, LLP 

30 Rockefeller Plaza 

New York. NY 10112-0228 

US 


Telephone No. 
(212) 750-5000 


Facsimile No. 
(212) 750-5020 


Teleprinter No. 


1 1 Mark this check-box where no agent or common representative is has been appointed and the space above is used instead to 
1 — 1 indicate a special address to which correspondence should be sent. 



Form PCT/RO/IOI (first sheet) (January 1997; reprint January* 1998) Legaisiar isse. Form pctreq 
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I 



Sheet No. 



Continuation of Box No. Ill FURTHER APPLICANTS AND/OR (FURTHER) INVENTORS 


If none of the following sub-boxes is used, this sheet is not to be included in the request 


Name and address: (Family name followed by given name; for a legal entity. Jull official designation. 
The address mt^t include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (i.e. country) of residence if no State of residence is indicated below.) 

KALVA. HARI 

Department of Electrical Engineering 

Columbia University 

500 West 120th Street 

New York. NY 10027-6699 

US 


This person is: 

1 1 applicant only 

IXI ^plicant and inventor 

1 1 inventor onlv (If this check-box 
is marked do not fill in below.) 


State (i.e. country) of nationality: 
IN 


State (le. country) of residence: 
US 


pis person is applicant 1 | all designated | | all designated Stales except fCT the United States | — 1 the States indicated in 
for the purposes of: 1 — 1 States 1 — I the Unrted States of Amenca \gA of America only 1 1 the Supplemental Box 


Name and address :^ami5' name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (i.e. country) of residence if no State of residence is indicated below.) 


This person is: 

1 1 applicant only 

1 1 applicant and inventor 

1 j inventor onlv (If this check-box 
is marked, do not fill in below.) 


State (i.e. country) of nationality: 


State (i.e. country) of residence: 


This person is applicant | 1 all designated 1 1 all designated States except | 1 the United States 1 — 1 the States indicated in 
for the purposes of: 1 — 1 States 1 — 1 the United States of America 1 1 of America only 1 1 the Supplemental Box 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State (i.e. country) of residence if no State of residence is indicated below.) 


This person is: 

1 1 applicant only 

1 1 ^plicant and inventor 

1 1 inventor onlv flf this check-box 
is marked, do not fill in below.) 


State (i.e. country) of nationality: 


State (i.e. country) of residence: 


This person is applicant 1 1 all designated 1 1 all designated States except | 1 the United States 1 1 the Slates indicated in 
for the purposes of: 1 — 1 States 1 1 the United States of America 1 1 of Amenca only 1 1 the Supplemental Box 


Name and address: (Family name followed by given name; for a legal entity, full official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant's State fi.e. country) of residence if no State of residence is indicated below.) 


This person is: 

1 1 applicant only 

1 1 applicant and inventor 

1 1 inventor onlv (If this check-box 
is marked, do not fill in below.) 


State (i.e. country) of naiionalit>': 


State (i.e. country) of residence: 


This person is applicant 1 1 all designated 1 1 all designated States except 1 1 the United Stales 1 1 the States indicated in 
for the purposes of: 1 — 1 States I — j the United States of America 1 1 of America only 1 1 the Supplemental Box 


1 1 Further applicants and/or (further) inventors are indicated on another continuation sheet. 



Form PCT/RO/IOI (continuation sheet) (Januao' 1997: reprint Januar>- 1998) '^^^'^'^^ ^"""^ ''^^'^^^ See Notes to the request fo^ 



Box No.V DESIGNATION OF STATES 



^^^Nol^ 



Sheet No. . ..3. 



□ 



The following designations are hereby made under Rule 4.9(a) (mark the applicable check-boxes; at least one must be marked): 
Regional Patent 

GM Gambia, KE Kenya, LS Lesotho. MW Malawi, SD Sudan. SZ Swaziland 
Uganda, ZW Zimbabwe, and any other State which is a Contracting State of the Harare Protocol and of the PCT 

K-i" 15*'"" ».','n"*= AM Annenia, AZ Azerbaijan. BY Belarus, KG Kyrg>'zstan, KZ Kazakhstan. MD Republic of 
Moldova, RU Russian Federation. TJ Tajikistan, TM Turkmenistan, and any other State which is a Contraclinc State of 
the Eurasian Patent Convention and of the PCT v-uuuucung oiaie oi 

K EP European Patent: AT Ausbia, BE Belgium. CH and LI Switzerland and Liechtenstein, DE Germany. DK Denmark, ES 
Spam, FI Fmland. FR Franw. GB United Kingdom, GR Greece, IE Ireland, IT Italy. LU Luxembourg MC 
V^SiS^t'f.^.^Xf^na^'^^ ' ""^ ^^'"-^ * Contracting State of the Eufopea^ 

□ OA OAPI Patent BFBurkinaFaso. BJ Benin. OF Central African Republic. CG Congo. CI Cote d'lvoire. CM Cameroon, 
GA Gabon. GN Guinea, ML Mali, MR Mauntania, NE Niger. SN Senegal, TD Chad, TG Togo, and any other State 
which IS a member State of OAPI and a Contracting State of the PCT (if other kind of protection or treatment des^ 
specify on dotted line) 

National Patent (if other kind of protection or treatment desired, specify on dotted line): 



□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 

□ 
□ 
□ 
□ 
□ 
□ 
n 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 

□ 
□ 
□ 

□ 
□ 
□ 
□ 
□ 
□ 



AL Albania 

AM Armenia 

AT Austria 

AU Australia 

AZ Azerbaijan 

BA Bosnia and Herzegovina 

BB Barbados 

BG Bulgaria 

BR Brazil 

BY Belarus 

CA Canada 

CH and LI Switzerland and Liechtenstein 

CN China 

CU Cuba 

CZ Czech Republic ; 

DE Germany 

DK Denmark 

EE Estonia 

ES Spain 

FI Finland 

GB United Kingdom 

GE Georgia 

GH Ghana 

GM Gambia 

GW Guinea-Bissau 

HU Hungary 

ID Indonesia 

IL Israel 

IS Iceland 

JP Japan 

KE Kenya 

KG Kyrgyzslan 

KP Democratic People's Republic of Korea 



□ 
□ 
□ 
□ 
□ 
□ 

□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 

□ 
□ 
□ 
□ 



LT Lithuania 
LU Luxembourg 
LV Latvia 

MD Republic of Moldova 

MG Madagascar 

MK The former Yugoslav Republic of Macedonia 



MN Mongolia 

MW Malawi 

MX Mexico 

NO Norway 

New Zealand 

Poland 

Portugal 

Romania 

Russian Federation .... 
Sudan 
Sweden 
Singapore 

Slovenia 

Slovakia 

Sierra Leone 

Tajikistan 

Turkmenistan 

Turkey 

Trinidad and Tobago 

Ukraine 

Uganda 

United States of America 



NZ 
PL 
PT 
RO 
RU 
SD 
SE 
SG 
SI 
SK 
SL 
TJ 
TM 
TR 
TT 
UA 
UG 
US 



UZ 
VN 
YU 



Uzbekistan 
Viet Nam 
Yugoslavia 



KR 
KZ 
LC 
LK 
LR 
LS 



Z\V Zimbabwe 

Republic of Korea i . . . ^ 

j,^,^. , . Check-boxes re$er\'ed for designatmg States (for the purposes of 

i^azaKHSian a national patent) which have become part>' to the PCT after 

1 issuance of this sheet: 



Saint Lucia 
Sri Lanka 
Liberia 
Lesotho 



□ 
□ 
□ 



n addition to the designations made above, the applicant also makes under Rule 4.9(b) all designations which would be permined 
under the PCT except the designation(s) of 

applicant declares that those additional designations are subject to confirmation and that any designation which is not confin^ed 
before the expiration of 15 momhs from the priorit>' date is to be regarded as withdrawn by the applicant at the expiration of that time 
limit. (Confirmation of a designation consists of the fling of a notice specifying that designation and the payment of the designation and 
confirmation fees. Confirmation must reach the receiving Office within the J 3-month time limit.) 



Form PCT/RO/IOl (second sheet) (January 1998) 



LegalStar 1998. Form PCTREO 
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I 



Sheet No. . . .f . 



I 



Box No. VI PRIORITY CLAIM 



Further priority claims are indicated in the Supplemental Box | J 



The priority of the following earlier application(s) is hereby claimed: 



Country 
(in which, or for which, the 
application was filed) 



item (1) 
US 



Filing Date 
( day/month/year) 



14 February 1997 
( 14-02-97) 



Application No. 



60/037.779 



Office of filing 
( onfyjor regional or 
international application) 



item (2) 



item (3) 



( ) ^ 

Mark the following check-box if the certified copy of the earlier application is to be issued by the Office which for the purposes of the 
present international application is the receiving Office (a fee may be required): 

(Vj The receiving Office is hereby requested to prepare and transmit to the International 
jQI Bureau a certified copy of the earlier application(s) identified above as item(s): 



Box No. VII INTERNATIONAL SEARCHING AUTHORITY 



Choice of International Searching Authority (ISA) (If two or more International Searching Authorities a 

are competent to carry out the international search, indicate the Authority chosen; the two-letter code may be used): 1 S A YUS 



Earlier search Fill in where a search (international, international-type or other) by the International Searching Authority has already been 
out or requested and the Authority is now requested to base the international search, to the extent possible, on the results of that earlier search 
such search or request either by reference to the relevant application (or the translation thereof) or by reference to the search request: 
Country (or regional Office): Date (day/month/year): Number- 



Box No. Vm CHECK LIST 



This international application contains 
the following number of sheets: 



1 . request 

2. description 

3. claims 

4. abstract 

5. drawings 

Total 



sheets 
sheets 
sheets 
sheets 
sheets 



sheets 



Thi s inte rnational aoDlication is accompanied by the item(s) marked below: 
CU of Attorney ^' Kl fee calculation sheet 



= □ 



copy of general 
power of attorney 



6. I I separate indications concerning 
*— J deposited microorganisms 



statement explaining j j j nucleotide and/or amino acid 



lack of signature " i — I sequence listing (diskette) 

□ Sl'7XfJ.yi «• Kl other r.p.c,M Transmittal Letter 

as item(s): 



Figure No. 



of the drawings (if any) should accompany the abstract when it is published. 



Box No. DC SIGNATURE OF APPLICANT OR AGENT 



obvious from ^riading%e^re^^^ ^^^^ person signing and the capacity in which the person signs (if such capacity is not 



Peter A. Businger (Ao^nt) 




■ For receiving Office use only — — — 

1. Date of actual receipt of the purported 
international application: 


2. Drawings: 


3, Corrected date of actual receiot due to later but 
timely received papers or drawings completing the 
purported international application: 


1 1 received: 


4. Date of timely receipt of the required 
corrections under PCT Article 1 1(2): 


[ 1 not received: 


5. International Searching Authonrs' . , | r i 1 -r™™ « , ^ ^ . , 

specified bv the applicant: ' ISA/ fl search copy delayed 

1 ' — ' until search fee is paid 




Date of receipt of the record copy 
by the International Bureau: 



Form PCT/RO/101 (last sheet) (Januar>' 1994; reprint Januao' 1998) Legaistar 1998. Form PCTREQ 
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This s^^^ not %srt of and does not count as a sheet of the i^^^ation^ application. 



PCX 



FEE CALCULATION SHEET 
Annex to the Request 



Applicant's or agent's 
file reference 



30919-PCT 



For receiving Office use only 



International application No. 



Dale stamp of the receiving Office 



Applicant 

THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK 



CALCULATION OF PRESCRIBED FEES 
1. TRANSMITTAL FEE 



240.00 



US 



450.00 



2. SEARCH FEE 

International search to be carried out by 

(If two or more International Searching Authorities are competent in relation to the international 
application, indicate the name of the Authority which is chosen to carry out the international search.) 

3. INTERNATIONAL FEE 
Basic Fee 

The international application contains 

first 30 sheets 

0 X 



sheets. 



455.00 



remaining sheets additional amount 



0.00 



b2 



Add amounts entered at b j and b ^ md enter total at B 
Designation Fees 

The international application contains 



• . L 



455.00 



B 



105.00 



. designations. 



420.00 



number of designation fees 
payable (maximum I J) 



amount of designation fee 



D 



Add amounts entered at B and D and enter total at I 

(Applicants from certain States are entitled to a reduction of 75% of the 
international fee. Where the applicant is (or all applicants are) so entitled, 
the total to be entered at I is 25% of the sum of the amounts entered at B and 



875.00 



ID 



4. FEE FOR PRIORITY DOCUMENT 

5. TOTAL FEES PAYABLE 

Add amounts entered at T, S, I and P, and enter total in the TOTAL box 



15.00 



1,580.00 



TOTAL 



I I The designation fees are not paid ai this time. 



MODE OF PAYMENT 

□ authorization to charge 
deposit account (see below) 

[Xl cheque 

postal money order 



I I bank draft 

I I cash 

□ revenue stamps 



I I coupons 

I I other (specify'): 



DEPOSIT ACCOUNT AUTHORIZATION (this mode of payment may not be available at all receiving Offices) 
The RO/ I I is hereby authorized to charge the total fees indicated above to my deposit account. 

f\7| is hereby authorized to charge anv deficiency or credit anv overpavment in the total fees indicated above to mv 
deposit account. ' - r- . 

□ is hereby authorized to charge the fee for preparation and transmittal of the priority document to the Internationa] 
Bureau of WIPO to my deposit account. 



02-4377 



► my deposit account. 
13 February 1998 



Deposit .Account Number 



Date (day/month/year) 



Signature 




Form PCT/RO/101 (Annex) (Januar>- 1996: reprint Januar>' 1998) Legaistar i9S8. Form pctrfee 



See Notes to the fee calculation sheet 



PCT/US98/02668 

PATENT COOPERATION TREATY 



From the INTERNATIONAL BUREAU 



PCT 


To: 


NU 1 IrlLA 1 lUN Ur cLbL 1 lUN 


United States Patent and Trademark 




vjTiice 


(PCT Rule 61.2) 


(Box PCT) 




Lrystai riaza / 




Washington, DC 20231 




ETATS-UNIS D'AIVIERIQUE 


Date of mailing (day/month/year) 


in its capacity as elected Office 


22 October 1998 (22.10.98) 




tntemational application No. 


Applicant's or agenf s file reference 


1 /UoUo/UZOOO 


30919-PCT 


International filing date (day/month/year) 


Priority date (day/month/year) 


13 February 1998(13.02.98) 


14 February 1997 (14.02.97) 


Applicant 




ELEFTHERIADIS, Alexandres et al 




1. The designated Office is hereby notified of its election made: 


1 X| in the demand filed with the International Preliminary Examining Authority on: 


09 September 1998 (09.09.98) 


1 1 in a notice effecting later election filed with the International Bureau on: 


2. The election | X| was 


1 


1 1 was not 


( 


made before the expiration of 19 months from the priority date or, where Rule 32 applies, within the time limit under 


Rule 32.2(b). 






Authorized officer 


The International Bureau of WlPO 




34, chemin des Colombettes 


J. Leitao 


1211 Geneva 20, Switzerland 




Facsimile No.: (41-22) 740.14.35 


Telephone No.: (41-22) 338.83,38 


Form PCT/IB/331 (July 1992) 


2299226 



